Method for device spectral sensitivity reconstruction

ABSTRACT

A method and system are provided for approximating spectral sensitivities of a particular image sensor, the image sensor having a color filter array positioned over the image sensor. In one example of the method, the method involves measuring spectral sensitivities of a set of image sensors each having a color filter array positioned over the image sensor, calculating mean spectral sensitivities of the set of image sensors for each color within the color filter array, measuring outputs of a particular image sensor when capturing a picture of a plurality of color patches under a first illuminant and calculating spectral sensitivities of the particular image sensor using the mean spectral sensitivities and the output of the particular image sensor. In some embodiments, the method further comprises utilizing the calculated spectral sensitivities to determine outputs of the particular image sensor under a second illuminant. In some embodiments, the method further comprises utilizing the calculated spectral sensitivities to calibrate a camera including the image sensor.

FIELD OF THE INVENTION

The present invention relates to digital cameras. More particularly, thepresent invention relates to calibrating digital cameras using measuredstatistical data of a set of cameras.

BACKGROUND OF THE INVENTION

The quality of an image captured by a color imaging system primarilydepends on three factors—sensor spectral sensitivity, illumination andobject scene. Illumination is important to be known. However, thespectral sensitivity characteristics are critical to the success ofimaging applications, and are necessary for the optimal design of theimaging system under practical constraints. Ultimately, the human visualsystem subjectively judges image quality.

A digital still camera (DSC) or video camera (camcorder) has a sensorthat is covered by a color filter array (CFA) to create pixel locations.A DSC typically uses red, green and blue (RGB) filters to create theirimage. Most current camcorders typically use cyan, magenta, yellow andgreen (CMYG) filters for the same purpose.

A conventional sensor is a charge-coupled device (CCD) or complimentarymetal oxide semiconductor (CMOS). An imaging system focuses a scene ontothe sensor and electrical signals are generated that correspond to thescene colors that get passed through the colored filters. Electroniccircuits amplify and condition these electrical signals for each pixellocation and then digitize them. Algorithms in the camera then processthese digital signals and perform a number of operations needed toconvert the raw digital signals into a pleasing color image that can beshown on a color display or sent to a color printer.

Each color camera has a unique sensor, CFA, and analog electronicssystem. The sensor and CFA have part-to-part variations. Accordingly,the electronic system needs to be calibrated for each camera. The goalis to make a “real world” scene captured with different cameras look thesame when rendered on a display device. In order to calibrate anindividual camera, the properties of the individual camera's primarycolor channels (CMYG for a camcorder; RGB for a DSC) need to be measuredso that the individual camera's response to known colors can bequantized.

Color cameras require multiple classes of sensors with differentspectral sensitivities. By placing CFA's in series with sensors, usuallyon a pixel-by-pixel basis, such multiple classes can be created. Whenthe color filters are placed in a mosaic pattern, one color per pixel,the cameras are referred to as CFA cameras.

Cameras rely on illumination to capture information of an object scene.Cameras are utilized and calibrated under a variety of illuminants.Estimating how information obtained from under one illuminant wouldtransform under another illuminant is a challenge.

Sensor characteristics, which consist of electronic sensor, color filterand optical lens, are the critical parts in the design of digital colorcameras. A camera is the input end in a color input-output system. Thus,its capability to acquire precise signals under a noisy environment canmake significant contributions to the processing and output imagequality.

More and more attention is being paid to spectral-based approaches,where the input signal and the output signal are all treated as aspectral power distribution. For example, the object surface reflectanceis captured spectrally, and rendered or printed spectrally to match theoriginal surface characteristics.

Spectral sensitivity may be obtained via direct measurement of aparticular camera output. The spectral sensitivity functions can then beused to determine the mapping relationship between device output signalsand object color perception values for any samples under any interestedilluminant. Direct measurement of spectral sensitivities with aspectroradiometer and monochromator gives rather accurate results. It iswell known how a digital camera can be calibrated if the SS curve foreach camera is known. Unfortunately, directly measuring the SS curvefrom a set of cameras is both time-consuming and expensive.

SUMMARY OF THE INVENTION

What is needed is an improved system having features for addressing theproblems mentioned above and new features not yet discussed. Broadlyspeaking, the present invention fills these needs by providing a methodand system for camera spectral sensitivity (SS) reconstruction fromcapturing a picture under a known illuminant of a set of known colorpatches. It should be appreciated that the present invention is able tobe implemented in numerous ways, including as a method, a process, anapparatus, a system or a device. Inventive embodiments of the presentinvention are summarized below.

In one embodiment, a method is provided for approximating spectralsensitivities of an image sensor, the image sensor having a color filterarray positioned over the image sensor. The method comprises measuringan output of the image sensor when capturing a picture of a set of colorpatches under a first illuminant and calculating spectral sensitivitiesof the image sensor based on the measured output.

In another embodiment, a method is provided for approximating spectralsensitivities of a particular image sensor, the image sensor having acolor filter array positioned over the image sensor. The methodcomprises measuring spectral sensitivities of a set of image sensorseach having a color filter array positioned over the image sensor,measuring outputs of a particular image sensor when capturing a pictureof a plurality of color patches under a first illuminant and calculatingspectral sensitivities of the particular image sensor using the outputof the particular image sensor and the spectral sensitivities of the setof image sensors.

In a further embodiment, a method is provided for approximating spectralsensitivities of a particular image sensor, the image sensor having acolor filter array positioned over the image sensor. The methodcomprises measuring spectral sensitivities of a set of image sensorseach having a color filter array positioned over the image sensor,calculating mean spectral sensitivities of the set of image sensors foreach color within the color filter array, measuring outputs of aparticular image sensor when capturing a picture of a plurality of colorpatches under a first illuminant and calculating spectral sensitivitiesof the particular image sensor using the mean spectral sensitivities andthe output of the particular image sensor.

In still another embodiment, a method is provided for approximating aspectral sensitivity curve of a particular image sensor, the imagesensor having a color filter array positioned over the image sensor. Themethod comprises measuring spectral sensitivities of a set of imagesensors each having a color filter array positioned over the imagesensor, calculating mean spectral sensitivities of the set of imagesensors for each color within the color filter array, calculatingstandard deviation of the spectral sensitivities of the set of imagesensors for each color within the color filter array, measuring outputsof a particular image sensor when capturing a picture of a plurality ofcolor patches under a first illuminant, dividing the mean spectralsensitivities into intervals and calculating a spectral sensitivitycurve for each interval using the mean spectral sensitivities, thestandard deviation and outputs of the particular image sensor, under afirst illuminant.

In yet another embodiment, an image capturing system is provided havinga color filter array positioned over an image sensor. The imagecapturing system comprises an image sensing module to detect an inputlight, wherein the imaging sensing module includes a color filter arrayhaving a spectral sensitivity, a processing module coupled to the imagesensing module, wherein the processing module is configured to measurespectral sensitivities of a set of image sensors each having a colorfilter array positioned over the image sensor, calculate mean spectralsensitivities of the set of image sensors for each color within thecolor filter array, to measure outputs of a particular image sensor whencapturing a picture of a plurality of color patches under a firstilluminant, to calculate spectral sensitivities of the particular imagesensor using the mean spectral sensitivities and the outputs of theparticular image sensor.

In still another embodiment, a processing module of a particular camerais provided, having a color filter array positioned over an imagesensor. The processing module is configured to measure spectralsensitivities of a set of image sensors each having a color filter arraypositioned over the image sensor, calculate mean spectral sensitivitiesof the set of image sensors for each color within the color filterarray, to measure outputs of a particular image sensor when capturing apicture of a plurality of color patches under a first illuminant, tocalculate spectral sensitivities of the particular image sensor usingthe mean spectral sensitivities and the outputs of the particular imagesensor.

The invention encompasses other embodiments are configured as set forthabove and with other features and alternatives.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be readily understood by the followingdetailed description in conjunction with the accompanying drawings. Tofacilitate this description, like reference numerals designate likestructural elements.

FIG. 1 is a color filter array (CFA), in accordance with an embodimentof the present invention;

FIG. 2 shows sample SS curves for the CFA of FIG. 1, in accordance withan embodiment of the present invention;

FIG. 3 is a sample reflectance R curve, in accordance with an embodimentof the present invention;

FIG. 4 is a sample illuminance L curve, in accordance with an embodimentof the present invention;

FIG. 5A is an average SS curve for one color of a CFA, in accordancewith an embodiment of the present invention;

FIG. 5B shows an offset SS curve 504, which is the average SS curve ofFIG. 5A after being offset within each interval to approximate the realmeasurement within each interval, in accordance with an embodiment ofthe present invention; and

FIG. 6 illustrates a block diagram of an exemplary image capturingsystem configured to operate according to the reconstruction method, inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

An invention for a method and system for camera spectral sensitivity(SS) reconstruction from a limited number of known color patches isdisclosed. Numerous specific details are set forth in order to provide athorough understanding of the present invention. It will be understood,however, to one skilled in the art, that the present invention may bepracticed with other specific details.

The method involves constructing SS curves for a camera system includinga color filter array positioned over an image sensor so that an outputof the image sensor can be approximated under different illuminants. Themethod applies to both digital still cameras (DSCs) and to video cameras(camcorders), even though the two different types may use differentcolor filter arrays (CFAs) to generate their images. The two main typesof color filter arrays are RGB for DSCs and CMYG for video camcorders.This method of modeling the SS curves works equally well with either CFAtechnique. Note, however, that the RGB system is generally easier toimplement because the RGB system has fewer parameters that requireestimation. Thus, for simplicity of explanation, equations anddiscussions will be directed toward an RGB system, yet will beapplicable to both an RGB system and a CMYG system.

All the internal specifications of a camera's digital-processingpipeline do not have to be known in order to implement this method.However, the reconstruction method does assume that a user (e.g., amanufacturer) of the method desires to map known input color patchesthat can be measured under tightly controlled conditions to specifyoutput channel values. Color patches are commonly used as known input inthe process of calibrating a camera's output. A color patch is placedoutside of the camera in front of the lens. The user will vary theparameters of signal flow within the camera until the desired mappinghas the optimum effect.

To perform the mapping, the reconstruction system needs to know thespecific RGB values that a specific image sensor under a specificilluminant will produce. A “real world” image can take place under manydifferent illuminants. From an incandescent illuminant to a very blueilluminant in the afternoon, the RGB values will vary strongly accordingto the illuminant for the same patch. Ideally, to predict the cameradigital RGB values, the SS curve is known for each camera. For a knownilluminant, a known color patch with a known reflectance and known SScurve, the RGB values can be determined. Once an image sensor's SScurves are known, then under any illuminant, the RGB values associatedwith any patch can be predicted. Additionally, patches that have notbeen captured by the specific image sensor can be predicted if theircharacteristic reflectance spectrum has been measured.

The method of the present invention involves taking a picture with aspecific image sensor of a set of known color patches under a knownilluminant. The output of the image sensor is then measured and thisdata is utilized to calculate and predict the output of the image sensorunder other illuminants and other conditions.

FIG. 1 is a CFA 102, in accordance with an embodiment of the presentinvention. In a DSC or camcorder, there is a CFA positioned over theimage sensor. Every pixel in the image sensor is covered with some colorfrom the CFA. The CFA 102 of FIG. 1 has colors of red, green and blue.Each filter has a property called spectral sensitivity (SS).

FIG. 2 shows sample SS curves for the CFA of FIG. 1, in accordance withan embodiment of the present invention. The SS curves are a function ofthe wavelength A. Each color has its own spectral sensitivity curve,including a blue SS curve 202, green SS curve 204 and red SS curve 206.

FIG. 3 is a sample reflectance R curve 302, in accordance with anembodiment of the present invention. Each object in a scene has areflectance. Reflectance R is a function of the wavelength λ.

FIG. 4 is a sample illuminance L curve 402, in accordance with anembodiment of the present invention. Illuminance L is a function of thewavelength π.

The camera measures an electrical signal from an image sensor. D_(A)^(k) is an electrical output for one of the colors (red, green and blue)of the CFA and for each color patch K under a first illuminance A. Notethat, in practice, this calculation is performed for each color of theCFA. For example, the image sensor analysis for RGB will have three ofthese D_(A) ^(k) calculations, an analysis for each color: red; green;and blue. A spectral sensitivity estimate needs to be estimated for eachcolor (red, green and blue) of the CFA. As will be further explainedbelow, an object is to be able to calculate an accurate output signal ofthe particular image sensor for each color under a different illuminant.Calculations for only one color are described herein for simplicity ofexplanation. However, in practice, the calculations described herein areperformed for each color of the CFA.

The reconstruction method does not involve taking time consumingmeasurements of all the variables. In other words, the reconstructionsystem does not take measurements from pictures under varyingilluminances (e.g., daylight, indoors, night).

For a particular camera, which has it's own particular image sensor, thereconstruction system captures a picture of the plurality of colorpatches under an illuminant. At the beginning of the calibration processor other development and correction process, the reconstruction systemmeasures a certain number of image sensors with a spectrometer to obtainSS output curves of the image sensors within the set. These measurementsprovide statistics which are used in the output prediction process.Next, the reconstruction system estimates the spectral sensitivity ofthe particular image sensor. Then, the reconstruction system predictsthe image sensor output with a different illuminance B using thefollowing equation: $\begin{matrix}{D_{B}^{k} = {\int_{\lambda_{0}}^{\lambda_{N}}{{B(\lambda)}{{SS}(\lambda)}{R^{k}(\lambda)}{{\mathbb{d}\lambda}.{Electrical}}\quad{output}\quad{signal}\quad D\quad{under}\quad{illuminance}\quad{B.}}}} & {{Eq}.\quad 1}\end{matrix}$In Eq. 1, B(λ) is illuminance and is a function of wavelength λ. Thevalue SS is the spectral sensitivity property for one color of a sensor.FIG. 2 is an example of this SS. The value R is the reflectance of lightcoming from the color filter. Reflectance R is a function of thewavelength λ. For the color patch k (k=1, . . . ,K) with the reflectanceR^(k), the image sensor output D_(B) ^(k) is calculated. The functionsare multiplied and the integral is taken. The reconstruction systemcalculates the image sensor output D_(B) ^(k) for each color of thecolor filter array by knowing all these variables, the illuminance B(λ),the spectral sensitivity SS(λ) and the reflectance R(λ), which are allfunctions of the wavelength λ.

The method involves taking K measurements for each color of the colorfilter array. The goal is to reconstruct K outputs for illuminant B.Some statistics are known. For example, assume that it is known thatthere are M image sensors. Preferably, M image sensors provide adequaterepresentation of all image sensors in a product line. The followingequations are useful to the calculations: $\begin{matrix}{{{SS}(\lambda)} \cong {\overset{\_}{{SS}(\lambda)} + \left\{ {{\begin{matrix}{{\Delta\quad{SS}_{1}{\sigma^{0}(\lambda)}},} & {\lambda_{0} \leq \lambda \leq \lambda_{1}} \\{{\Delta\quad{SS}_{2}{\sigma^{0}(\lambda)}},} & {\lambda_{1} \leq \lambda \leq \lambda_{2}} \\\vdots & \quad \\{{\Delta\quad{SS}_{N}{\sigma^{0}(\lambda)}},} & {\lambda_{N - 1} \leq \lambda \leq \lambda_{N}}\end{matrix}.{Approximation}}\quad{of}\quad{SS}\quad{for}\quad{each}\quad{RGB}\text{}{color}\quad{of}\quad{the}\quad{CFA}\quad{on}\quad N\quad{spectral}\quad{{intervals}.}} \right.}} & {{Eq}.\quad 2}\end{matrix}$Approximating SS, as in Eq. 2, involves first calculating very quicklyand simply the mean function SS(λ) according to the following equation:$\begin{matrix}\begin{matrix}\begin{matrix}{\overset{\_}{{SS}(\lambda)} = {\frac{1}{M}{\sum\limits_{i = 1}^{M}{{{SS}_{i}(\lambda)}.}}}} \\{{Mean}\quad{spectral}\quad{sensitivity}\quad{function}\quad{for}\quad{each}\quad{RGB}}\end{matrix} \\{{color}\quad{of}\quad{the}\quad{CFA}\quad{obtained}{\quad\quad}{from}{\quad\quad}M\quad{sample}\quad{{sensors}.}}\end{matrix} & {{Eq}.\quad 3}\end{matrix}$The reconstruction system calculates a mean function for each color,including red, green and blue. In other words, each color has its ownspectral sensitivity curve. According to Eq. 2, approximating SS alsoinvolves calculating the standard deviation σ⁰(λ) of the spectralsensitivity for the image sensor set. The following equation defines thestandard deviation: $\begin{matrix}{{\sigma^{0}(\lambda)} = {\frac{1}{\sigma_{\max}}{\sqrt{\frac{1}{M}{\sum\limits_{i = 1}^{M}\left( {{{SS}_{i}(\lambda)} - \overset{\_}{{SS}(\lambda)}} \right)^{2}}}.{Normalized}}\quad{standard}\quad{{deviation}.}}} & {{Eq}.\quad 4}\end{matrix}$One example where the standard deviation is useful: if a particularsensor has a factory related problem, the SS curve may deviate up, andthe standard deviation variable takes such an occurrence intoconsideration. Because it is squared, the standard deviation is anabsolute variable.

The method uses the foregoing equations and statistics to calculatespectral sensitivities for a particular image sensor. For explanatorypurposes, the following description describes the spectral sensitivityreconstruction for only one color within the color filter array.However, the following description generally applies to the other colorsof the RGB system.

FIG. 5A is an average SS curve 502 for one color of a CFA, in accordancewith an embodiment of the present invention. This average SS curve 502has been divided into N intervals. Assume the spectral sensitivity SS(λ)for a particular image sensor is the mean spectral sensitivity SS(λ)plus the standard deviation σ⁰(λ) multiplied by a coefficient ΔSS, as inEq. 2. This calculation will happen for each interval in order toapproximate the entire SS curve.

FIG. 5B shows an offset SS curve 504, which is the average SS curve 502of FIG. 5A after being offset within each interval to approximate thereal measurement within each interval, in accordance with an embodimentof the present invention. The integral of the whole range is the sum ofthe integrals of each range.

Notice that the SS curve is no longer smooth because each interval is anapproximation. A goal is to have the integral of this approximation beequal to D_(A) ^(k). Instead of taking actual measurements, eachmeasurement is approximated as the integral of an average SS plus thestandard deviation multiplied by a coefficient, all done over Nintervals. The following linear equation defines this approximation:$\begin{matrix}{\begin{matrix}{D_{A}^{k} = {\int_{\lambda_{0}}^{\lambda_{N}}{{A(\lambda)}{{SS}(\lambda)}{R^{k}(\lambda)}{\mathbb{d}\lambda}}}} \\{\cong {{\int_{\lambda_{0}}^{\lambda_{N}}{{A(\lambda)}\overset{\_}{{SS}(\lambda)}{R^{k}(\lambda)}{\mathbb{d}\lambda}}} +}} \\{{\Delta\quad{SS}_{1}{\int_{\lambda_{0}}^{\lambda_{1}}{{A(\lambda)}{\sigma^{0}(\lambda)}{R^{k}(\lambda)}{\mathbb{d}\lambda}}}} + \ldots +} \\{\Delta\quad{SS}_{N}{\int_{\lambda_{N - 1}}^{\lambda_{N}}{{A(\lambda)}{\sigma^{0}(\lambda)}{R^{k}(\lambda)}{{\mathbb{d}\lambda}.}}}}\end{matrix}{{Linear}\quad{equations}\quad{approximating}\quad{spectral}\quad{{sensitivity}.}}} & {{Eq}.\quad 5}\end{matrix}$The coefficient ΔSS is a constant and can therefore be taken outside ofthe integral. For the sake of simplicity, the output from the imagesensor having an average spectral sensitivity is as follows:$\begin{matrix}{{\int_{\lambda_{0}}^{\lambda_{N}}{{A(\lambda)}\overset{\_}{{SS}(\lambda)}{R^{k}(\lambda)}{\mathbb{d}\lambda}}} = {{G_{A}^{k}.{Output}}{\quad\quad}{of}\quad{camera}\quad{having}\quad{average}\quad{spectral}\quad{{sensitivity}.}}} & {{Eq}.\quad 6}\end{matrix}$Also for simplicity, the standard deviation for one interval from Eq. 5is defined as follows: $\begin{matrix}{{\int_{\lambda_{n - 1}}^{\lambda_{n}}{{A(\lambda)}{\sigma^{0}(\lambda)}{R^{k}(\lambda)}{\mathbb{d}\lambda}}} = {{F_{n}^{k}.{Standard}}\quad{deviation}\quad{for}\quad{each}\quad{{interval}.}}} & {{Eq}.\quad 7}\end{matrix}$The method of the present invention involves taking a picture with aspecific camera having a color filter array positioned over an imagesensor of a set of known color patches under a known illuminant. Theoutput of the image sensor is then measured and this data is utilized tocalculate and predict the output of the camera under other illuminantsand other conditions. The reconstruction system calculates the averageoutput and the standard deviations from each statistics set for eachcolor patch. Each average output from each statistics set for each colorpatch will be different. Next, the method involves solving the followingsystem of linear equations: $\begin{matrix}\begin{matrix}{{{\Delta\quad{SS}_{1}F_{1}^{1}} + {\Delta\quad{SS}_{2}F_{2}^{1}} + \cdots + {\Delta\quad{SS}_{N}F_{N}^{1}}} = {D_{A}^{1} - G_{A}^{1}}} \\\begin{matrix}{{{\Delta\quad{SS}_{1}F_{1}^{2}} + {\Delta\quad{SS}_{2}F_{2}^{2}} + \cdots + {\Delta\quad{SS}_{N}F_{N}^{2}}} = {D_{A}^{2} - G_{A}^{2}}} \\\begin{matrix}\vdots \\{{{\Delta\quad{SS}_{1}F_{1}^{K}} + {\Delta\quad{SS}_{2}F_{2}^{K}} + \cdots + {\Delta\quad{SS}_{N}F_{N}^{K}}} = {D_{A}^{K} - G_{A}^{K}}} \\{{{System}\quad{of}\quad{linear}\quad{equations}\quad{of}\quad{camera}}\quad} \\{{outputs}\quad{for}\quad K\quad{color}\quad{{patches}.}}\end{matrix}\end{matrix}\end{matrix} & {{Eq}.\quad 8}\end{matrix}$If the number of color patches equals the number of intervals, then thereconstruction system is able to find an exact solution of the system oflinear equations. It is necessary to find the coefficients ΔSS. Forexample, given measurements taken from 100 image sensors, thereconstruction system is able to calculate the standard deviation σ⁰(λ)in advance. The output of the image sensor having average spectralsensitivity G (from Eq. 8 and Eq. 6) is known. Also, the electricaloutput signal D (from Eq. 8) is a known measurement. It is thennecessary to calculate the coefficients ΔSS for each color patch. Thereare K equations for K color patches, and N coefficients for N intervals.If N equals K, the number of variables equals the number of equations.The reconstruction system can then solve the system of linear equationsrepresented in Equation 8. If N is not equal to K, it is possible tofind an approximate solution.

In Equation 8, note that the more intervals that the reconstructionsystem uses, the closer the system of equations provides a solution thatis closer to the real spectral sensitivity SS. Seven color patches maybe a practical number of color patches to use in the measurements andcalculations under normal conditions. Accordingly, statisticalinformation is used to approximate the spectral sensitivity SS.

In other words, the reconstruction system involves measuring the imagesensor output under known conditions and a known illuminant and thenusing statistical information to approximate the spectral sensitivity ofthe image sensor.

As shown in FIG. 5A, the mean spectral sensitivity is broken down intointervals. As shown in FIG. 5B, for each interval, differentcoefficients ΔSS are used to move standard deviation curves at eachinterval up or down a certain offset of the mean spectral sensitivitycurve in order to match the particular image sensor's output. With eachnew image sensor, a picture of the color patches under a knownilluminant is taken. The system of linear equations is setup toapproximate the output of the new image sensor as an output from anaverage image sensor plus some variations at different intervals.Spectral sensitivity intervals are an arbitrary length and can beadjusted as needed to get an accurate result. For example, a largeinterval can be used where the spectral sensitivity SS does not changemuch; a small interval can be used where the spectral sensitivity SSchanges a large amount.

FIG. 6 illustrates a block diagram of an exemplary image capturingsystem 601 configured to operate according to the reconstruction method,in accordance with an embodiment of the present invention. The imagecapturing system 601 is any device capable of capturing an image orvideo sequence, such as a camera or a camcorder. The image capturingsystem 601 includes imaging optics 602, an image sensing module 604, aprocessing module 606, a memory 608, and an input/output (I/O) interface610.

The imaging optics 602 include any conventional optics to receive aninput light representative of an image to be captured, to filter theinput light, and to direct the filtered light to the image sensingmodule 604. Alternatively, the imaging optics 602 do not filter theinput light. The image sensing module 604 includes one or more sensingelements to detect the filtered light. Alternatively, the image sensingmodule 604 includes a color filter array to filter the input light andone or more sensing elements to detect the light filtered by the colorfilter array.

The memory 601 can include both fixed and removable media using any oneor more of magnetic, optical or magneto-optical storage technology orany other available mass storage technology. The processing module 606is configured to control the operation of the image capturing system601. The processing module 606 is also configured to perform thereconstruction method described above. The I/O interface 610 includes auser interface and a network interface. The user interface can include adisplay to show user instructions, feedback related to input usercommands, and/or the images captured and processed by the imaging optics602, the image sensing module 604, and the processing module 606. Thenetwork interface 610 includes a physical interface circuit for sendingand receiving imaging data and control communications over aconventional network.

Advantageously, the method performed by the reconstruction systeminvolves approximating the spectral sensitivity for a camera with acolor filter array positioned over an image sensor. It should also beunderstood that this method is able to be used with only an imagesensor, outside of a camera system. The reconstruction system usesmeasurements taken with one illuminant to then calculate the imagesensor's output under other illuminants. In other words, thereconstruction system uses one set of data from the image sensor under afirst illuminant to then calculate a new set of data under otherilluminants. In contrast, methods of the prior art teach taking newmeasurements to obtain the image sensor output under a new illuminant.Such a measuring process is unduly time consuming and costly. In thereconstruction system of the present invention, on the other hand, thenumber of measurements needed to calculate an image sensor's outputunder a different illuminant is substantially reduced. The expensivemeasuring process of the prior art is thereby avoided.

Method and System Implementation

Portions of the reconstruction system may be conveniently implementedusing a conventional general purpose or a specialized digital computeror microprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art.

The invention may also be implemented by the preparation of applicationspecific integrated circuits or by interconnecting an appropriatenetwork of conventional component circuits, as will be readily apparentto those skilled in the art.

The reconstruction system includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the reconstruction system. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical disks, DVD, CD-ROMS, micro-drive, and magneto-optical disks,ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices(including flash cards), magnetic or optical cards, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), thereconstruction system includes software for controlling both thehardware of the general purpose/specialized computer or microprocessor,and for enabling the computer or microprocessor to interact with a humanuser or other mechanism utilizing the results of the reconstructionsystem. Such software may include, but is not limited to, devicedrivers, operating systems, and user applications. Ultimately, suchcomputer readable media further includes software for performing thereconstruction method, as described above.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the reconstruction method, including but not limited to,calculating mean spectral sensitivities of the set of image sensors eachhaving a color filter array positioned over the image sensor,calculating standard deviation of the spectral sensitivities of the setof image sensors for each color within the color filter array, measuringoutputs of a particular image sensor when capturing a picture of aplurality of color patches under a first illuminant, calculating thespectral sensitivity of the particular image sensor using the meanspectral sensitivities, the standard deviation and the output of theparticular image sensor, and utilizing the calculated spectralsensitivity to determine outputs of the particular image sensor under asecond illuminant, according to processes of the reconstruction method.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

1. A method of approximating spectral sensitivities of an image sensor,the image sensor having-a-color filter array positioned over the imagesensor, the method comprising: measuring an output of the image sensorwhen capturing a picture of a set of color patches under a firstilluminant; and calculating spectral sensitivities of the image sensorbased on the measured output.
 2. The method of claim 1, furthercomprising utilizing the calculated spectral sensitivities to determinea calculated output of the image sensor under a second illuminant. 3.The method of claim 2, further comprising utilizing the calculatedspectral sensitivities and the calculated output under the secondilluminant to calibrate a camera including the image sensor.
 4. Themethod of claim 1, further comprising utilizing the calculated spectralsensitivities to calibrate a camera including the image sensor.
 5. Amethod of approximating spectral sensitivities of a particular imagesensor, the image sensor having a color filter array positioned over theimage sensor, the method comprising: measuring spectral sensitivities ofa set of image sensors each having a color filter array positioned overthe image sensor; measuring outputs of a particular image sensor whencapturing a picture of a plurality of color patches under a firstilluminant; and calculating spectral sensitivities of the particularimage sensor using the output of the particular image sensor and thespectral sensitivities of the set of image sensors.
 6. The method ofclaim 5, further comprising utilizing the calculated spectralsensitivities to determine outputs of the particular image sensor undera second illuminant.
 7. The method of claim 6, further comprisingutilizing the calculated spectral sensitivities and the output under thesecond illuminant to calibrate a camera including the image sensor. 8.The method of claim 5, further comprising utilizing the calculatedspectral sensitivities to calibrate a camera including the image sensor.9. The method of claim 5, wherein calculating spectral sensitivitiescomprises calculating mean spectral sensitivities of the set of imagesensors for each color within the color filter array.
 10. The method ofclaim 5, wherein the measured spectral sensitivities are functions of awavelength.
 11. The method of claim 10, wherein the plurality of colorpatches includes K color patches, wherein calculating spectralsensitivities comprises solving a system of linear equations comprisingdividing the mean spectral sensitivity curve into N intervals of thewavelength range.
 12. The method of claim 11, wherein K does not equalN.
 13. The method of claim 11, wherein K equals N.
 14. The method ofclaim 9, wherein calculating mean spectral sensitivities comprises:measuring spectral sensitivities of each image sensor in the set ofimage sensors; and averaging the spectral sensitivities of each imagesensor in the set of image sensors to obtain the mean spectralsensitivities.
 15. The method of claim 5, wherein the set of imagesensors includes M image sensors, wherein M image sensors provideadequate representation of image sensors in a product line.
 16. A methodof approximating spectral sensitivities of a particular image sensor,the image sensor having a color filter array positioned over the imagesensor, the method comprising: measuring spectral sensitivities of a setof image sensors each having a color filter array positioned over theimage sensor; calculating mean spectral sensitivities of the set ofimage sensors for each color within the color filter array; measuringoutputs of a particular image sensor when capturing a picture of aplurality of color patches under a first illuminant; and calculatingspectral sensitivities of the particular image sensor using the meanspectral sensitivities and the output of the particular image sensor.17. The method of claim 16, further comprising utilizing the calculatedspectral sensitivities to determine outputs of the particular imagesensor under a second illuminant.
 18. The method of claim 17, furthercomprising utilizing the calculated spectral sensitivities and theoutput under the second illuminant to calibrate a camera including theimage sensor.
 19. The method of claim 16, further comprising utilizingthe calculated spectral sensitivities to calibrate a camera includingthe image sensor.
 20. The method of claim 16, wherein calculatingspectral sensitivities comprises calculating standard deviation of thespectral sensitivities of the set of image sensors for each color withinthe color filter array.
 21. The method of claim 20, wherein the measuredspectral sensitivities are functions of a wavelength.
 22. The method ofclaim 21, wherein the plurality of color patches includes K colorpatches, wherein calculating the spectral sensitivities comprisessolving a system of linear equations comprising dividing the meanspectral sensitivity curve into N intervals of the wavelength range. 23.The method of claim 22, wherein K does not equal N.
 24. The method ofclaim 22, wherein K equals N.
 25. The method of claim 24, whereincalculating the spectral sensitivities of the particular image sensorcomprises: calculating an average output based on an integral of themean spectral sensitivity; calculating an offset at each interval basedon the standard deviation at each interval multiplied by a coefficientat each interval, wherein the coefficient at each interval is associatedwith a color patch; and adding the average output at each interval tothe offset at each interval, wherein the adding is performed at all Kintervals.
 26. The method of claim 25, wherein calculating a spectralsensitivity curve for each interval comprises adding an integral of themean spectral sensitivity at each interval to the standard deviationmultiplied by a coefficient for each interval, wherein the coefficientfor each interval is associated with a color patch.
 27. The method ofclaim 16, wherein calculating mean spectral sensitivities comprises:measuring spectral sensitivities of each image sensor in the set ofimage sensors; and averaging the spectral sensitivities of each imagesensor in the set of image sensors to obtain the mean spectralsensitivities.
 28. The method of claim 27, wherein the set of imagesensors includes M image sensors, wherein M image sensors provideadequate representation of image sensors in a product line.
 29. A methodof approximating a spectral sensitivity curve of a particular imagesensor, the image sensor having a color filter array positioned over theimage sensor, the method comprising: measuring spectral sensitivities ofa set of image sensors each having a color filter array positioned overthe image sensor; calculating mean spectral sensitivities of the set ofimage sensors for each color within the color filter array; calculatingstandard deviation of the spectral sensitivities of the set of imagesensors for each color within the color filter array; measuring outputsof a particular image sensor when capturing a picture of a plurality ofcolor patches under a first illuminant; dividing the mean spectralsensitivities into intervals; and calculating a spectral sensitivitycurve for each interval using the mean spectral sensitivities, thestandard deviation and outputs of the particular image sensor, under afirst illuminant.
 30. The method of claim 29, further comprisingcombining each interval of the spectral sensitivity curve to obtain acalculated spectral sensitivity curve for the particular image sensor.31. The method of claim 29, wherein the measured spectral sensitivitiesare functions of a wavelength.
 32. The method of claim 31, wherein theplurality of color patches includes K color patches, wherein dividingthe mean spectral sensitivities into intervals comprises dividing themean spectral sensitivities into N intervals of the wavelength range.33. The method of claim 32, wherein K does not equal N.
 34. The methodof claim 32, wherein K equals N.
 35. The method of claim 32, whereincalculating the spectral sensitivity curve for each interval comprisesadding the mean spectral sensitivities at each interval to the standarddeviation multiplied by a coefficient at each interval, wherein thecoefficient at each interval is associated with a color patch.
 36. Themethod of claim 29, wherein calculating mean spectral sensitivitiescomprises: measuring spectral sensitivities of each image sensor in theset of image sensors; and averaging the spectral sensitivities of eachimage sensor in the set of image sensors to obtain the mean spectralsensitivities.
 37. The method of claim 36, wherein the set of imagesensors includes M image sensors, wherein M image sensors provideadequate representation of image sensors in a product line.
 38. Acomputer-readable medium carrying one or more instructions forapproximating outputs of a particular image sensor under varyingilluminants, the image sensor having a color filter array positionedover the image sensor, wherein the one or more instructions, whenexecuted by one or more processors, cause the one or more processors toperform: measuring spectral sensitivities of a set of image sensors eachhaving a color filter array positioned over the image sensor;calculating mean spectral sensitivities of the set of image sensors foreach color within the color filter array; measuring outputs of aparticular image sensor when capturing a picture of a plurality of colorpatches under a first illuminant; and calculating spectral sensitivitiesof the particular image sensor using the mean spectral sensitivities andthe output of the particular image sensor.
 39. The computer-readablemedium of claim 38, wherein the one or more instructions cause the oneor more processors to further perform utilizing the calculated spectralsensitivities to determine outputs of the particular image sensor undera second illuminant.
 40. The computer-readable medium of claim 39,wherein the one or more instructions cause the one or more processors tofurther perform utilizing the calculated spectral sensitivities and theoutput under the second illuminant to calibrate a camera including theimage sensor.
 41. The computer-readable medium of claim 38, wherein theone or more instructions cause the one or more processors to furtherperform utilizing the calculated spectral sensitivities to calibrate acamera including the image sensor.
 42. The computer-readable medium ofclaim 38, wherein calculating spectral sensitivities comprisescalculating standard deviation of the spectral sensitivities of the setof image sensors for each color within the color filter array.
 43. Animage capturing system having a color filter array positioned over animage sensor, the image capturing system comprising: an image sensingmodule to detect an input light, wherein the imaging sensing moduleincludes a color filter array having a spectral sensitivity; aprocessing module coupled to the image sensing module, wherein theprocessing module is configured to measure spectral sensitivities of aset of image sensors each having a color filter array positioned overthe image sensor, calculate mean spectral sensitivities of the set ofimage sensors for each color within the color filter array, to measureoutputs of a particular image sensor when capturing a picture of aplurality of color patches under a first illuminant, to calculatespectral sensitivities of the particular image sensor using the meanspectral sensitivities and the outputs of the particular image sensor.44. The image capturing system of claim 43, wherein the processingmodule is further configured to utilize the calculated spectralsensitivities to determine outputs of the particular image sensor undera second illuminant.
 45. The image capturing system of claim 44, whereinthe processing module is further configured to utilize the calculatedspectral sensitivities and the output under the second illuminant tocalibrate a camera including the image sensor.
 46. The image capturingsystem of claim 43, wherein the processing module is further configuredto utilize the calculated spectral sensitivities to calibrate a cameraincluding the image sensor.
 47. The image capturing system of claim 43,wherein calculating spectral sensitivities comprises calculatingstandard deviation of the spectral sensitivities of the set of imagesensors for each color within the color filter array.
 48. A processingmodule of a particular camera having a color filter array positionedover an image sensor, wherein the processing module is configured tomeasure spectral sensitivities of a set of image sensors each having acolor filter array positioned over the image sensor, calculate meanspectral sensitivities of the set of image sensors for each color withinthe color filter array, to measure outputs of a particular image sensorwhen capturing a picture of a plurality of color patches under a firstilluminant, to calculate spectral sensitivities of the particular imagesensor using the mean spectral sensitivities and the outputs of theparticular image sensor.
 49. The processing module of claim 48, whereinthe processing module is further configured to utilize the calculatedspectral sensitivities to determine outputs of the particular imagesensor under a second illuminant.
 50. The processing module of claim 49,wherein the processing module is further configured to utilize thecalculated spectral sensitivities and the output under the secondilluminant to calibrate a camera including the image sensor.
 51. Theprocessing module of claim 48, wherein the processing module is furtherconfigured to utilize the calculated spectral sensitivities to calibratea camera including the image sensor.
 52. The processing module of claim48, wherein calculating spectral sensitivities comprises calculatingstandard deviation of the spectral sensitivities of the set of imagesensors for each color within the color filter array.